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The  purpose  of  this  study  was  to  adapt  and  illustrate  the  use  of  a 
computer  program  to  score  binary  patterns  of  response  on  a  short- 
form  predictor  test  (Electronics  Technician  Selection  Test  and  the 
General  Classification  Test)  so  as  to  maximize  the  correlation 
between  this  predictor  and  a  criterion  (the  final  school  grade  in  the 
Basic  Electronics  and  Electricity  School). 
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I.  INTRODUCTION 


The  Navy  has  been  very  much  interested  in  recent  years  in  the 
possibility  of  using  short- form  tests  to  reduce  testing  time  while 
maintaining  or  even  increasing  test  reliability  and  validity. 

The  advantages  of  a  short- form  test  are  manifold.  With  a 
short  but  reliable  and  valid  test  the  Navy  could  save  thousands  of 
dollars  in  training  costs  by  weeding  out,  before  training  even  began, 
those  individuals  svho  would  probably  not  succeed.  The  administration 
of  the  test  could  be  done  at  a  training  command,  e.  g. ,  Naval  Training 
Center,  San  Diego,  Bair.bridge,  etc. .  or  even  by  a  recruiter.  For 
example,  if  an  individual  desires  to  be  a  radioman  and  talks  to  a 
recruiter  about  joining  the  Navy  only  if  accepted  for  radioman  training, 
it  would  be  advantageous  for  both  die  service  and  that  individual  if  a 
brief  test  of  possibly  five  to  seven  minutes’  duration  could  be  admin¬ 
istered,  graded  and  evaluated  on  the  spot  against  the  individual's 
desires  foi  such  a  Navy  career.  With  this  brief  test  both  the  Navy 
and  the  potential  recruit  would  know,  in  a  relatively  short  period 
of  time,  vhether  the  man  would  succeed  in  radioman  training. 
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II.  BACKGROUND 


Moonan  (Ref.  1)  pioneered  this  type  of  work  for  the  Navy  by 
constructing  a  computer  program  having  the  capability  of  identifying 
combinations  of  test  items  that  have  maximal  validity.  This  program, 
entitled  SEQUIN  (an  acronym  for  Sequential  Item  Nominator)  first 
selects  an  item  that  has  highest  validity  with  the  criterion.  The  pro¬ 
gram  then  continues  to  select  another  item  which,  when  combined 
with  the  first,  produces  a  two-  item  test  with  a  higher  validity  than 
any  other  two-item  test  that  includes  the  first  item.  This  process 
continues  until  the  required  number  of  items  is  selected  and  the 
maximum  validity  tor  this  number  of  items  is  obtained.  The  advan¬ 
tage  of  such  a  program  is  that  a  fairly  long  test,  such  as  the  General 
Classification  Test  (GCT),  might  be  shortened  without  sacrificing 
validity  while  test  time  might  be  significantly  reduced. 

SEQUIN  has  shown,  repeatedly,  that  a  short- form  test  is  at 
least  as  predictive  of  final  school  grade  as  its  long- form  counterpart 
(Ref.  2).  Swanson  and  Rimland  (Ref.  3)  have  found  that  a  short  form 
of  the  GCT,  e.g.  one- half  to  one- third  of  the  original  length,  is  even 
more  predictive  of  recruit  final  achievement  (RFAT)  than  the  complete 


form. 


111.  THE  PROBLEM 


This  study  attempts  to  increase  further  the  predictive  validity 
of  an  already  brief  test.  The  method,  developed  by  Dr.  R.  A. 
Weitzman  of  the  Naval  Postgraduate  School,  Monterey,  Californio, 
is  to  weight  item  responses  so  as  to  maximize  the  coi relation  with  the 
criterion. 

On  an  n-item  test  where  each  question  is  graded  to  he  either 
correct  or  incorrect,  there  are  2n  different  possible  patterns  of 
correct  and  incorrect  responses.  Thus,  for  example,  on  a  five-item 
test  there  are  32  possible  pattern  scores  as  opposed  to  six  possible 
scores  if  just  the  number  of  correct  responses  were  tallied. 

For  example,  suppose  a  three  question  test  is  given  to  a  group 
of  recruits  in  an  attempt  to  predict  their  success  in  a  Navy  training 
school.  There  are  eight  (2°)  combinations  of  patterns  running  from 
000  to  111  (where  zeros  are  incorrect  responses  and  ones  are  correct). 
A  subject  having  a  pattern  of  101  has  the  same  number  correct  as 
another  subject  with  the  pattern  110,  that  is,  two  out  of  three. 

However,  the  first  individual's  score  might  be  more  predictive  of 
success  in  a  particular  training  school  than  the  second  subject’s 
binary  pattern. 

This  study  will  focus  on  four  different  tests  or  test  scores, 


defined  as  follows: 


1.  Predictor  -  the  predictor  is  a  long- form  test  used  for 
predicting  success  in  a  Navy  training  school.  In  this  study,  the 
predictor  is  th--'  Electronics  Technician  Selection  Test  or  the  General 
Classification  Test.  Scores  on  the  predictor  are  determined  by 
counting  the  items  answered  c  rrrectly. 

2.  Criterion  -  the  final  school  grade  in  the  Basic  Electronics 
and  Electricity  School. 

3.  Total  Correct  -  the  rota),  number  of  correct  responses  out 
of  hie  seven  questions  selected  by  SEQUIN  analysis  for  this  study. 

4.  Pattern  Score  -  a  special  score  assigned  to  each  pattern  of 
responses  on  the  same  seven  items  used  to  compute  total  correct. 

(A  precise  definition  of  pattern  scores  will  be  given  in  Section  VE. ) 

Thus,  me  purpose  of  this  study  was  to: 

1.  Gather  large  pools  of  data  from  a  Navy  training  school. 

2.  Extract  several  suitable  questions  from  the  General  Classi¬ 
fication  Test  (GCT)  and  the  Electronic  Technician  Selection  Test 
(ETST), 

3.  Write  o  computer  program  that : 

a.  constructs  all  possible  patterns  of  ones  and  zeros  for 
die  number  of  extracted  questions 

b.  calculates  pattern  scores  for  each  individual  pattern 

c.  assigns  pattern  scores  to  subjects 

d.  correlates  the  pattern  scores  of  the  subjects  with 
theii  final  school  grades 
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e.  correlates  the  standard  predictor  test  scores  (either 
GCT  or  ETST)  with  final  school  grades 
I.  correlates  total  correct,  with  final  school 
grades 

g.  correlates  pattern  score  with  total  correct 
n.  calculates  a  multiple  correlation  coefficient  between 
a  combination  of  pattern  scores  and  total  correct  and 
final  school  grades 

i.  calculates  test  statistics  for  the  correlations 

j.  calculates  regression  weights  for  predicting  final 
school  grades  from  total-correct  scores 

k.  creates  a  frequency  distribution  showing  number  of 
subjects  with  each  pattern  score 

l.  outputs  all  information  in  an  easy- to- read  form  for 
use  in  future  studies 

4.  Determine  those  patterns  indicative  of  success  for  a  parti¬ 
cular  training  school, 

5.  Test  pattern-  score  predictions  by  suitable  cross-validation. 


IV.  DATA 


All  data  used  in  this  research  were  obtained  from  Mr.  Leonard 
Swanson  of  the  Naval  Personnel  and  Training  Research  Laboratory, 

San  Diego,  California,  and  were  stored  on  nine- track  magnetic  tape 
(Ref.  4).  The  data  consisted  of  the  individual  records  of  approxi¬ 
mately  2400  trainees  who  started,  but  not  necessarily  finished,  the 
Navy  Bar,ic  Electronics  and  Electricity  School  in  San  Diego.  Each 
trainee’s  record  consisted  of  the  equivalent  of  six- computer  card 

i 

records  listing  such  information  as: 

1.  Responses  to  items  on  the  GCT,  ETS'> ,  and  Arithmetic 
Aptitude  Test  (ARl) 

2.  Scores  on  the  GCT,  ARI,  and  ETST 

3.  Navy  serv-ce  number 

4.  Enlisted  rating 

5.  Final  school  grade  in  Basic  Electronics  and  Electricity 
School 

Tests  used  as  predictors  included  the  GCT  and  ETST.  The  GCT 
consists  of  60  verbal  analogies  and  40  sentence- completion  items 
with  a  35- minute  time  limit.  The  ETST  consists  of  three  separately 
timed  sections:  math  with  20  items  and  a  25- minute  time  limit; 
science  with  20  items  and  a  15- m:  ’ute  time  limit;  and  electricity  and 
radio  with  30  items  and  a  120-  minute  time  limit  (Ref.  5). 
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Two  sets  of  questions  were  provided  by  Mr.  Swanson  along 
with  their  answer  keys.  The  first  set  of  question?,  consisting  of 
seven  GCF  items,  and  the  second  set,  consisting  of  seven  ETST 
Hems,  were  selected  using  the  SEQUIN  program.  The  p- values, 
question  types  and  item  validities  are  shown  in  Appendixes  A  anc  B. 

The  criterion  consisted  of  final  school  grades  in  the  Basic 
Electronics  and  Electricity  School. 

A.  DATA  PREPARATION 

Two  programs  were  written  to  extract  and  put  into  usable  forms 
all  pertinent  data  for  the  study.  (A  glossary  of  terms  used  in  all 
programs  is  contained  in  Appendix  L. )  The  first  program  checked 
for  completeness  of  an  individual's  record,  i.e. ,  the  presence  of  six 
computer- card  images,  and  rejected  those  subjects  whose  files  were 
deficient,  unfortunately  several  records  contained  special  characters, 
e,g. ,  dashes,  asterisks,  etc.,  instead  of  integers.  Therefore,  the 
first  data  preparation  program  converted  any  of  these  special 
characters  to  zeros.  Thus,  a  response  other  than  an  integer  from  one 
to  five  was  changed  to  a  zero  and  counted  as  an  incorrect  response. 

If  a  needed  score  such  as  the  GCT,  ETST  or  final  school  grade  was 
blank  or  contained  some  non- numerical  mark,  the  record  for  that 
individual  was  rejected  as  being  incomplete.  (It  is  possible  that  some 
of  those  incomplete  records  resulted  from  subjects  not  finishing  the 
school,  i.e. ,  being  required  to  leave  die  service  because  of  physical. 


emotional,  or  academic  problems. )  The  output  from  this  program  was 
written  on  tape  or  data  cell  and  on  paper. 

Appendix  C  is  the  flowchart  of  this  first  program.  Appendix  D 
is  the  program  listing. 

Although  a  subject’s  record  consisted  of  six  computer- card 
records,  most  information  was  superfluous.  Of  the  six  cards,  data 
from  three,  at  most,  were  considered.  Using  the  answer  key  supplied 
by  Mr.  Swanson,  the  second  program  graded,  on  different  occasions 
those  E  rST  or  GCT  questions  under  consideration.  Specifically,  it 
assigned  a  value  of  one  to  a  correct  response  and  a  value  of  zero  to 
an  incorrect  response.  By  assigning  ones  and  zeros  to  the  responses, 
the  binary  pattern  was  formed.  The  program  also  read  the  criterion 
score  and  the  predictor  score. 

The  output  from  tills  second  program  consisted  of: 

1.  binary  pattern 

2.  criterion  score 

3.  predictor  score 

4.  an  in-house  identification  number 

5.  tne  subject's  service  number 

Appendix  E  is  the  flowchart  for  this  second  pregram.  Appendix  F  is 
the  program  listing. 


V,  THE  MODEL 
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Hie  main  program  is  divided  into  several  distinct  sections: 
reading  of  data,  determination  of  a  joint  frequency  distribution, 
computation  of  pattern  scores,  assignment  of  pattern  scores  to  sub¬ 
jects,  computation  of  correlation  coefficients  (r’s),  computation  of 
test  statistics  for  r  differences,  construction  of  response  patterns, 
ordering  of  response  patterns  according  to  the  scores  computed  for 
them,  calculation  of  a  multiple  correlation  coefficient,  calculation  of 
the  correlation  coefficient  between  patter,,  scores  and  total  correct 
construction  of  a  frequency  distribution  showing  the  number 
of  people  with  each  pattern  score,  and  output  (printed  and 
punched). 

A  complete  listing  of  the  program  is  presented  in  Appendix  G. 

A.  DOUBLE  PRECISION  REQUIREMENT 

Because  of  the  large  sample  sizes  and  the  relatively  large 
magnitude  of  several  parameters,  it  was  necessary  to  use  double 
precision  floating  point  numbers. 

B,  THE  DATA  CARD 

The  data  card  initializes  four  variables  that  are  frequently  used 
in  counting  Icops  (EX)  loops).  The  variables,  Nl,  N2,  N3,  and  N4, 
represent,  respectively,  "the  number  of  people  in  the  sample,  number 


of  elements  in  a  pattern,  the  range  of  criterion  scores,  and  the 
number  of  possible  binary  combinations  using  N2  items  (2^2). 

C.  READING  THE  DATA 

Data  is  read  in  only  a  prescribed  format.  For  this  program, 
the  individual’s  data  record  card  is  set  up  as  shown  in  Table  I. 

There  are  two  read  statements.  One  read  statement  carries 
out  the  reading  of  data  that  is  to  be  used  in  the  computations  of  the 
program.  The  other  road  statement  reads  a  dummy  variable,  ”1DUM. 
By  placing  the  read  statement  involving  1DUM  before  or  following  the 
main  read  statement  (involving  binary  pattern,  criterion,  predictor, 
etc. ),'  control  over  alternate  selections  of  data  can  be  attained.  For 
example,  if  odd  numbered  data  were  only  to  be  considered,  the  read 
statement  involving  IDUM  would  follow  the  main  read  statement  thus 
acting  as  a  dummy  procedure  to  control  data  input.  Note  that  all 
input  data  is  in  FORTRAN  ”  1  format." 

The  variable  "j"  is  used  as  the  EX)  LOOP  counter  involving 
personnel  with  only  one  exception.  That  exception  is  in  the  determi¬ 
nation  of  the  joint  frequency  distribution.  An  "1"  DO  L.OOP  is  used 
for  all  other  counting  operations. 

At  this  point,  the  total  correct  out  of  the  extracted  questions 
is  calculated.  The  "E”  array  stores  this  information.  This  array  is 
used  in  the  calculation  of  the  sum  of  total  correct  for  all  subjects  and 
the  sum  of  the  squares  of  total  correct  for  all  subjects.  This  infor¬ 
mation  is  later  used  in  the  computation  of  correlation  coefficients 


TABLE  I 


RECORD  DATA  CARD  SETUP  FOR 
VALIDATION  AND  CROSS-VALIDATION  PROGRAMS 


Column  Number 


8,9 


10,11 


12-15 


Item 

Program  Symbol 

Subject’s  binary 
pattern 

P(I,J) 

Criterion  Score 

C(J) 

Predictor 
(Score  on  ETST) 

AO)  ‘ 

.In-house  ident. 
number 

D(J) 

and  the  mean  and  standard  deviation  for  use  in  the  calculation  of  a 
test- retest  correlation  coefficient. 

D.  THE  JOINT  FREQUENCY  DISTRIBUTION 

A  joint  frequency  distribution  is  constructed  using  decimal 
equivalents  of  the  2^2  binary  patterns  and  the  range  of  criterion 
scores.  The  rows  of  the  matrix  (denoted  by  matrix  variable  F) 
represent  the  decimal  equivalents  of  the  binary  patterns,  and  in  this 
case  there  are  128  (2?)  binary  patterns  (the  reason  for  using  seven 
questions  is  explained  in  the  METHODS  section).  However,  the  lowest 
binary  pattern  score  (0000000)  is  also  equal  to  the  decimal  value  zero. 
Therefore,  a  value  of  one  is  added  to  all  decimal  equivalents.  In 
this  way  the  first  row  is  row  or.c,  not  zero,  and  the  last  row  is  row 
128. 

The  column  numbers  correspond  to  successive  criterion  scores. 
Column  one  of  the  matrix  corresponds  to  the  subjects’  lowest  crite¬ 
rion  score.  In  this  case,  the  lowest  criterion  score  was  30  and  the 
higest  was  99.  The  matrix  is  represented  in  Figure  1. 

"  The  "B"  array  is  used  to  store  the  decimal  equivalent  of  an 
individual’s  binary  pattern. 

E .  COMPUTATION  OF  PATTER N  SCORES 

The  pattern  score  for  a  pattern  is  the  average  score  of  subjects 
who  have  the  pattern  and  is  calculated  from  the  F  matrix  by  tallying 
the  number  of  subjects  having  the  pattern  and  each  criterion  score. 


20 


Figure  1 


CONSTRUCTION  OF  A  MATRIX  DESCRIBING  THE 
JOINT  FREQUENCY  DISTRIBUTION 


Columns** 

(30)  (99) 

1  69 


Rows* 


*Row  numbers  are  decimal  equivalents  of  binary  patterns  plus  one. 

**Column  numbers  are  criter  ion  scores  plus  one  minus  the  lowest 

criterion  score.  Numbers  in  parentheses  are  actual  criterion 


scores. 


This  number  is  multiplied  by  the  criterion  score  and  summed,  and 
the  sum,  SI,  is  divided  by  the  total  number  of  subjects  having  the 
pattern,  S2. 

If  any  of  the  128  patterns  is  not  used,  because  no  one  has  the 
pattern,  both  SI  and  S2  are  set  equal  to  zero,  and  an  arbitrary 
score  of  -1  is  assigned  to  the  pattern. 

Immediately  following  the  computation  of  all  pattern  scores, 
the  scores  are  outputted  on  punched  cards.  The  pattern  scores 
obtained  in  this  study  are  presented  in  Appendix  H. 

F.  ASSIGNMENT  OF  PATTERN  SCORE  TO  SUBJECTS 

A  subject’s  decimal  equivalent  to  his  binary  pattern  is  deter¬ 
mined,  and  he  is  assigned  the  pattern  score  for  that  decimal 
equivalent  (the  row  index  corresponding  to  i  ie  pattern  in  the  F 
matrix). 

G.  COMPUTATION  OF  CORRELATIONS 

Correlation  coefficients  are  then  calculated  between  the 
criterion  and  the  predictor  (GCT  or  ETST)  and  between  the  criterion 
and  the  assigned  pattern  scores. 

The  sums  of  criterion  scores  (Cl),  pattern  scores  (XI),  and 
predictor  scores  (Al)  are  determined  along  with  the  corresponding 
sums  of  squares  (C2,  X2,  A  2).  The  sum  of  the  products  of  the 
criterion  and  predictor  scores  (V),  as  well  as  the  criterion  and  pattern 
scores  (W),  is  also  determined.  The  correlation  coefficients  for 


pattern  vs.  criterion  (R2)  and  predictor  vs.  criterion  (Rl)  are  then 
calculated.  The  Z  test  statistic  for  the  difference  between  these  r's 
is  also  calculated. 

Three  other  correlation  coefficients  are  computed  later  in  the 
program:  a  multiple  correlation  coefficient  (see  I  below),  the 

ft 

correlation  coefficient  between  total-correct  and  pattern  scores,  and 
a  test- retest,  correlation  coefficient  used  as  an  estimate  of  the 
reliability  of  total  correct  scores  on  the  predictor. 

H.  CONSTRUCTION  OF  RESPONSE  PATTERNS 

Since  there  are  128  (2^)  different  patterns  of  responses  ranging 
from  0000000  to  illllll,  the  computer  is  assigned  the  otherwise 
tedious  and  difficult  job  of  constructing  and  outputting  these  patterns. 
A  difficulty  encountered  is  that  leading  zeros  of  various  binary 
patterns,  although  stored  without  incident  in  the  machine,  are  lost 
upon  printing.  Because  of  this,  all  zeros  in  the  binary  patterns  are 
converted  to  twos.  This  fact  is  noted  on  the  printed  output  (Appendix 
H). 

L  MULTIPLE  CORRELATION  COEFFICIENT 

The  multiple  ocrrelation  coefficient  indicates  the  strength  of 
relationship  between  one  variable  and  a  linear  combination  of  two  or 
more  others  that  produces  the  strongest  relationship.  Since  different 
predictor  variables  are  sometimes  intercorreiated  and  so  duplicate 
one  another,  the  multiple  correlation  coefficient  depends  on  the 


intercorrelation  of  different  predictor  variables  as  well  as  on  the 
correlation  of  each  with  the  criterion  variable  (Ref.  6). 

Specifically,  the  multiple  correlation  between  criterion  scores 
and  a  combination  of  pattern  scores  and  total- correct  scores  is 
computed. 

Since  the  coefficient  of  multiple  correlation  considers  the 
inter-relationship  between  the  predictor  variables,  it  should  have, 
theoretically,  a  greater  value  than  the  correlation  between’ either 
predictor  and  final  school  grades  alone. 

The  significance  of  the  multiple  r  is  next  computed  using  an  F 
statistic  where  F  is  the  ratio  of  the  variance  of  the  residuals  on  the 
criterion  before  considering  the  multiple  correlation  coefficient  and 
the  variance  of  the  residuals  after  consideration. 

J.  COMPUTATION  OF  REGRESSION  WEIGHTS 

Since  there  is  a  possibility  that  some  binary  patterns  will  not 
be  used  (i.e. ,  there  may  be  some  binary  patterns  no  one  has  because 
the  sample  size  is  small  in  relation  to  the  number  of  binary  combi¬ 
nations),  it  is  conceivable  that  an  individual  in  the  cross-validation 
group  might  have  a  pattern  that  no  one  in  the  validation  group 
has.  Correspondingly,  regression  weights  are  computed  from  the 
relation  between  total-correct  and  criterion  scores  in  the  validation 
group  that  are  to  be  used  as  input  in  the  cross-validation  study  to 
determine  scores  for  individuals  having  pattern  scores  equal  to  -1. 


K.  OUTPUTS  I  AND  II 

The  pattern  responses  are  then  sorted  according  to  pattern 
score  from  the  lowest  (fl)  to  the  highest  (73.66)  and,  in  conjunction 
with  the  pattern  score  and  total  correct  of  that  binary  pattern,  are 
printed  out  in  tabular  form.  The  table  and  results  thus  obtained  are 
shown  in  Appendix  H  as  Output  I. 

Next,  tables  are  prepared  listing  the  subject’s  in-house  identi-. 
fication.  b’sjpredictor  scor°.  his  final  school  grade  (criterion  score), 
the  pattern  score  associated  with  his  binary  pattern,  and,  finally, 
the  total  correct  scored  out  of  the  seven  questions.  A  sample  showing 
the  first  50  subjects  is  presented  in  Appendix  I  as  Output  H. 


mm 


L.  ADDITIONAL  OUTPUT 


IK; 

n 

HI: 


All  correlations  and  test  statistics  computed  during  the 


execution  of  the  program  are  also  printed.  These  results  are  pre¬ 
sented  and  discussed  in  the  RESULTS  section. 
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VI.  CROSS- VALIDATION 

■  mm,  n  . .  %  m  ■  ■  n  -  »  -*<■»  ■  .1 

Cross-validation  is  a  method  used  to  estimate  the  magnitude 
of  sampling  variation.  In  cross-validation,  results  are  obtained 
from  a  second  sample  of  people  for  comparison  with  the  results  of 
an  initial  sample.  If  the  results  obtained  from  the  second  sample 
confirm  the  results  of  the  first  the  results  are  said  to  hold  up  under 
cross-  validation. 

In  addition  to  the  validation  or  main  program,  described  in  the 
preceding  section,  this  study  makes  use  of  a  cross-validation  program, 
which  is  essentially  a  portion  of  the  main  program.  It  differs  in  that 
pattern  score  and  regression  weights  derived  from  the  previous 
program  are  read  in  with  new  subjects*  personal  data  and  that  patterns 
are  not  constructed,  pattern  scores  are  not  calculated,  and  there  is 
no  need  for  a  joint  frequency  distribution.  The  program  listing  for 
the  cross*  validation  is  presented  in  Appendix  J. 

As  can  be  seen  from  Appendix  H,  there  are  fourteen  binary 
patterns  that  were  not  used  by  the  validation  group  in  the  ETST  study. 
Therefore,  the  cross-validation  program  has  to  determine  if  a  subject 
has  a  pattern  that  was  not  used  in  the  validation  program  and,  if  he 
has,  it  must  assign  him  a  score  using  the  regression  weights  deter¬ 
mined  from  the  validation  group  and  his  total- correct  score. 


A.  ASSIGNMENT  OF  PATTERN  SCORES 


Various  other  methods  of  assigning  pattern  scores  to  patterns 
that  no  individual  in  the  validation  group  has  were  attempted.  These 
methods  included:  using  the  average  patten,  score  derived  from  the 
main  program,  weighting  more  heavily  those  patterns  appearing 
more  frequently  than  those  appearing  less  frequently,  ignoring  a 
subject  in  the  cross-validation  who  had  a  pattern  no  one  had  in  the 
validation  group  (with  adjustment  of  corresponding  variables,  e.  g. , 
sample  size),  and  finally  using  the  regression  weights. 

With  only  one  exception,  that  of  using  the  regression  weights, 
all  methods  of  attack  failed.  All  pattern-score  validities  were 
significantly  lowered  in  all  the  other  cases.  (The  reason  for  the 
abrupt  drop  in  pattern- score  correlation  coefficients  in  the  cross- 
validation  is  discussed  in  the  RESULTS  section! ) 

Using  the  regression  weights,  however,  pattern  score  validi¬ 
ties  maintained  a  maximum.  Scores  were  obtained  by  adding  the 
product  of  total  correct  and  the  slope  regression  weight  to  the 
regressed  mean. 

Inputs  for  the  cross-validation  consisted  of  the  same  information 
as  noted  in  the  main  program  plus  the  regression  weights  and  the 
pattern  scores  from  the  main  program. 

The  tabular  results  for  the  first  fifty  subjects  (even  numbers 
only)  is  presented  in  Appendix  K  as  Output  III. 
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VII.  METHOD 


The  validation  and  cross-validation  programs  were  first  used 
on  GCT  data.  Not  only  was  the  GCT  data  analyzed,  but  it  also 
served,  at  the  beginning  of  the  research  effort,  as  a  test  platform  for 
debugging  the  validation  and  cross-validation  computer  programs. 

The  study  concentrated  on  the  ETST  data,  however. 

A.  PRELIMINARY  STUDIES 

GCT  data  were  used  in  preliminary  studies.  Use  of  GCT  data 
as  a  predictor,  as  originally  planned,  was  unsatisfactory  because 
the  GCT  was  not  designed  as  a  predictor  of  success  in  a  training 
school  and,  of  the  seven  questions  considered  in  the  study,  approxi¬ 
mately  one- third  of  the  sample  subjects  had  all  correct,  which  is 
hardly  an  indication  of  predictive  validity. 

The  first  step  in  the  study  was  the  determination  of  the  sample 
size  to  be  used  in  the  validation  and  cross- valididation  programs. 
Since  the  total  number  of  possible  combinations  of  ones  and  zeros 
was  128  (2;),  it  was  decided  that  an  appropriate  sample  size  in  the 
main  program  would  be  2, 000.  This  would  result  in  the  theoretical 
utilization  of  15-16  subjects  per  binary  pattern: 

2, 000  subjects  _  _ 

128  patterns  "  / 


The  remaining  subjects  in  the  sample  (379)  would  then  be  used  in 
cross-validation  studies. 

Table  11  summarizes  the  results  of  this  first  effort.  As  can  be 
seen  from  these  results,  the  greatest  validity  for  the  validation 
(main)  group  was  obtained  from  the  predictor  vs.  criterion  scores 
(r  =  0. 51).  However,  the  relationship  between  the  pattern  and 
criterion  scores  was  only  0.44.  Although  lower  than  the  predictor 
validity  coefficient,  it  was  still  better  than  the  r  for  total  correct 
(0. 37).  The  high  absolute  values  of  the  test  statistics  indicate  that 
all  the  differences  were  significant. 

The  cross  validation  tells  essentially  the  same  story.  The 
validities  for  the  pattern  and  total  correct  were  very 
nearly  the  same  as  in  the  validation  program.  However,  the  validity 
for  the  patterns  fell  short  of  its  counterpart  in  the  validation  program 
(rXval  =  •  34  vs.  rvaj  =  .  44).  This  phenomenon  resulted  from  the 
weighting  of  item  responses  which  maximized  the  correlation  with 
the  criterion,  i.e. ,  minimized  the  error  of  prediction,  thereby 
capitalizing  on  chance  in  the  validation  group.  The  fact  that  chance 
played  an  important  role  in  the  validation  program  was  further 
illustrated  by  one  subject  who  had  a  binary  pattern  with  four  ones, 
i.  e. ,  four  out  of  seven  correct,  but  who  also  had  the  highest  of  all 
pattern  scores. 

Another  explanation  for  the  substantial  reduction  in  pattern 
vs.  criterion  validities  is  that  the  mean  pattern  score  was  used  for 
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TABLE  II 


CORRELATION  COEFFICIENTS  AND 
TEST  STATISTICS  DERIVED  FROM 
THE  GENERAL  CLASSIFICATION 
TEST 


Main 

Cross 

Validation 

■  r(pattera) 

0. 44 

0.34 

r(predictcr) 

0. 51 

0.52 

Z 

-2.72 

-3. 11 

r (total  ones) 

0.  37 

0. 36 

Z 

2. 74 

-.27 

NOTE:  1.  The  first  Z  is  for  die  difference  between  the  pattern- 

criterion  and  the  predictor- criterion  correlations.  The  second  Z 
is  for  the  difference  between  pattern- criterion  and  total  correct- 
criterion  correlations. 

2.  The  sample  size  in  the  main  study  was  2,000  subjects 
while  the  sample  size  in  the  cross-validation  study  was  379  subjects. 
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individuals  in  the  cross-validation  group  who  had  patterns  no  one  had 
in  the  validation  group. 

B.  ASSIGNMENT  OF  PATTERN  SCORES 

Because  of  the  discrepancies  in  pattern-  score  validities  for 
both  the  validation  and  cross-validation  programs,  the  problem  of 
assigning  a  valid  pattern  score  to  an  individual  who,  in  the  cross- 
validation  process,  had  a  pattern  no  one  had  in  the  main  validation 
arose.  Therefore,  the  several  approaches  mentioned  earlier  were 
formulated  and  attempted. 

1.  The  First  Solution 

The  first  of  these  proposed  solutions  involved  the  use  of 
weights  proportional  to  the  number  of  subjects  having  a  pattern. 

The  weights  were  to  be  calculated,  along  with  die  pattern  scores, 
in  the  main  program  and  outputted  on  punched  cards.  The  theory 
behind  this  solution  was  that  if  a  binary  pattern  appeared  very 
frequently  it  should  have  been  counted  more  heavily  in  the  cross- 
validation  than  those  patterns  appearing  less  frequently.  Once  again, 
considering  the  subject  who  had  the  highest  pattern  score  with  only 
four  correct,  it  would  appear  logical  that  that  person  was  not  typical 
and  should  not  have  been  counted  equally  as  others.  That  is,  would 
it  have  been  valid  to  give  his  score  the  same  weight  as  a  score 
that  was  25  per  cent  more  prevalent?  If  both  scores  receive  equal 
weight,  distortion  of  the  validities  must  certainly  occur.  Unfortunately, 
a  suitable  method  of  computing  and  applying  such  weights  was  not  found. 


2.  The  Second  Solution 


The  second  solution  was  to  reduce  the  number  of  questions 
used  in  the  study  from  seven  to  six.  With  only  six  questions  the 
number  of  binary  combinations  would  have  been  significantly  reduced 
(from  128  to  64)  resulting  in  the  utilization  of  more  binary  patterns. 

It  was  hoped,  in  fact,  that  all  binary  patterns  would  have  been  used. 
Thus,  when  going  into  the  cross-validation  phase  all  patterns  would 
have  had  pattern  scores  and  the  need  for  generating  pattern- score 
substitutes  in  the  cross-validation  could  have  been  eliminated. 
However,  e/en  with  consideration  of  only  six  questions  (64  combina¬ 
tions  of  ones  and  zeros),  eight  binary  patterns  were  not  used. 
Furthermore,  the  magnitudes  of  the  correlation  coefficients  decreased 
markedly.  Therefore,  this  approach  was  eliminated. 

3.  The  Third  Solution 

The  third  solution  called  for  die  elimination  of  those 
subjects  in  the  cross-validation  who  had  a  binary  pattern  no  one 
had  in  the  validation  study.  The  theory  behind  this  solution  was,  in 
essence,  to  eliminate  the  problem  by  pretending  it  wasn’t  there. 

This  solution  was  not  suitable  for  apparent  reasons.  For  a  test  to 
be  valid  in  a  real  environment,  vis  a  vis  a  laboratory  environment, 
it  must  consider  all  contingencies. 

4.  The  Final  Solution 

It  was  finally  decided  to  calculate  regression  weights  and 
use  these  in  assigning  pattern  scores  to  subjects  in  the  cross-valida¬ 
tion  who  had  patterns  no  one  had  in  die  main  program. 
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C.  READ  IN  OF  ALTERNATE  DATA 


The  possibility  of  sample  bias  was  also  considered,  e.  g. , 
predictor  or  criterion  scores  of  the  entire  sample  could  have  been 
placed  in  .  order  of  increasing  or  decreasing  magnitude.  Therefore, 
it  was  decided  to  split  the  sample  in  half;  the  first  half  to  be  use^. 
in  the  validation  program  and  the  second  half  in  the  cross-validation 
program.  The  main  or  validation  program  was  then  designed  to 
read  the  records  of  every  alteriiate  subject,  e.g. ,  every  odd- 
numbered  subject,  and  make  appropriate  calculations  from  those 

data.  The  cross-validation  also  read  every  alternate 
but  complementary  record.  Thus,  for  example,  if  the  main  program 
read  every  odd  record,  the  cross-validation  program  correspondingly 
read  every  even  record.  Unfortunately,  however,  splitting  the 
sample  this  way  resulted  in  a  drop  in  the  number  of 

subjects  per  binary  pattern  from  fifteen  to  approximately  nine. 

D.  THE  ETST  STUDY 

After  solving  the  problem  of  assigning  pattern  scores  to 
subjects  in  the  cross-validation  study,  the  research  focused  on 
utilization  of  the  ETST  as  the  predictor. 

Data  preparation  followed  the  same  procedures  as  those 
noted  in  the  data-preparation  section  of  this  thesis. 

In  addition  to  the  tables  and  correlation  coefficients  computed 
in  the  validation  and  cross-validation  processes,  the  programs  also 
outputted  the  sum  of  total  correct  and  the  sum  of  the  squares  of 


total  correct  for  all  subjects.  This  was  used  in. the  computation  of 
the  mean  and  variance  for  total  correct  (total  ones).  The  reason 

for  these  calculations  was  to  determine  the  test- retest  correlation 

coefficient. 

1.  Test- Retest  Reliability  Coefficient 

The  test- retest  reliability  coefficient,  as  described  by 
Weitzman  (Ref.  7),  is  an  estimate  of  the  correlation  between  identical 
versions  of  a  test  taken  by  the  same  persons  in  independent  trials. 

For  a  test  with  n-items  and  a- alternatives,  this  estimate  is: 

rtr  =  1  -  n-M 

aSZ-  (1) 

where  M  and  S  are  the  mean  and  standard  deviation,  respectively. 

This  estimate  of  the  test- retest  reliability  coefficient 
can  be  used  in  the  determination  of  the  correction  for  attenuation. 

2.  Correction  for  Attenuation 

Because  correlation  results  are  obtained  from  fallible 
measurements,  errors  tend  to  reduce  or  attenuate  the  correlation 
between  traits.  Using  the  formula  for  correction  for  attenuation,  it 
is  possible  to  estimate  what  the  correlation  would  be  if  perfect, 
errorless  measurements  were  available  (Ref.  8).  Correlation 
coefficients  that  are  corrected  for  attenuation  cannot  be  used  in  pre-  • . 
diction  equations  but  can  be  used  when  analyzing  relationships  to 
make  allowances  for  random  errors  of  measurement. 

Using  the  test- retest  correlation  coefficient  computed  for 

*  *  V 

the  predictor,  it  is  possible  to  calculate  the  validity  of  the  predictor 


corrected  for  attenuation.  The  value  obtained  from  the  following 
formula  is  the  theoretical  correlation  coefficient  if  the  predictor 
were  error- free: 

r  r  = 

-VFS  (2) 

The  correlation  coefficient  rpQ  is  the  measured  vaPdity  between 
tlie  predictor  and  criterion,  and  ipp  is  the  test-  retest  reliability 
described  in  the  previous  section.  A  comparison  between  the 
validity  coefficient  (rp^)  and  the  validity  coefficient  corrected  for 
attenuation  (r^,  q)  was  used  as  an  indication  of  how  close  this  study 
came  to  the  theoretical  limit  of  validity  for  the  predictor.  Specifi¬ 
cally,  r  (total  correct  vs.  final  school  grade)  was  compared  to 
the  corresponding  correlation  coefficient  corrected  for  attenuation. 


VHI.  RESULTS 

A.  DETERMINATION  OF  LINEARITY 

A  product- moment  correlation  coefficient  is  good  only  if  a 
linear  relationship  exists  between  the  variables  that  are  being  corre¬ 
lated.  Figures  2  and  3  are  scatter  diagrams  which  were  used  to 
determine  if  a  linear  relationship  existed  between  total  correct  out 
of  seven  and  final  school  grade  and  the  full  ETST  score  and  the 
final  school  grade.  Note  that  almost  all  the  points  can  be  enclosed 
in  an  oval  which  goes  from  the  lower  left  to  the  upper  right,  there¬ 
fore  indicating  linearity  (Ref.  9). 

B.  CORRELATION  COEFFICIENTS  AND  TEST  STATISTICS 
Table  III  lic.cs  the  values  for  all  test  statistics  and  correlation 

coefficients. 

As  can  be  seen  from  that  table,  the  value  of  rpattern  score 

decreases  from  0.76  in  the  validation  program  to  0.72  in  the 
cross-validation  program,  the  reduction  due  to  maximization  of 
chance  in  the  main  program.  The  computation  of  the  multiple 
correlation  coefficient  was  desired  to  see  if  pattern  scores  add  to 
die  predictive  ability  of  total- correct  scores.  The 
multiple  correlation  coefficient  did  not  increase  the  value  of 
rtotal  correct  t^lus  indicating  no  additional  predictive  ability'.  The 
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FINAL  SCHOOL  GRADE  (CRITERION) 


FIGURE  3 


30 _ 

35  50  55  60  65 


PATTERN  SCORE  CAS  A  PREDICTOR)' 

SCATTER  PLOT 


TABLE  III 


CORRELATION  COEFFICIENTS  AND 
TEST  STATISTICS  DERIVED  FROM 
THE  ELECTRONICS  TECHNICIAN 
SELECTION  TEST 


Main 

Cross- 

Validation 

r(pattern  score) 

"03  H 

0.72 

r(predictor) 

0.61 

0.60 

Z 

7.13 

4.93 

r( total  correct) 

0.72 

.  0.73 

Z 

2. 26 

-.54 

r(pattern  score/ 

total  correct) 

0.95 

0.95 

r(multiple) 

0. 76 

0. 73 

F 

176. 36 

15.38 

r(  test- retest) 

0.73 

Correction  for 

Attenuatior 

0. 85 

NOTE:  1.  The  first  Z  is  for  the  difference  between  the  pattern- 

criterion  and  die  predictor- criterion  correlations.  The  second  Z 
is  for  the  difference  between  pattern- criterion  and  tot?.!  correct- 
criterion  correlations. 

2.  The  sample  sizes  in  both  the  main  and  cross-validation 


studies  was.!  ,182  subjects,. 


large  value  of  F  indicates  that  the  total- correct  scores  contributed 
significantly  to  the  predictive  ability  of  the  pattern  scores, 
however. 

The  high  value  of  the  correlation  coefficient  between  pattern 
scores  and  total- correct  scores  indicated  that  the  seven  items  used 
in  the  study  constituted  a  very  valid  test  and  that  the  total  correct 
could  be  used  as  a  predictor  that  is  as  good  as  the  pattern  scores 
for  these  items. 

The  correction  for  attenuation  revealed  that  the  highest  validity 
theoretically  obtainable  by  improving  e  reliability  of  the  seven- 
item  predictor  was  0. 85,  The  value  actually  obtained,  0. 73,  was 
equal  to  the  test- retest  reliability  of  the  test.  Since  it  is  not  reason¬ 
able  to  expect  that  a  test  will  correlate  more  highly  with  another 
test  than  it  does  with  itself,  it  is  no  wonder  that  the  pattern  scores 
did  not  correlate  better  with  the  criterion  than  the  total- correct  did. 


IX.  CONCLUSION 


The  two  FORTRAN’  computer  programs  developed  in  this 
study  successfully  determined  and  correlated  pattern  scores  with 
the  criterion.  However*  the  questions  extracted  from  the  ETST 
were  so  highly  valid  that  they  could  have  been  used  alone,  1.  e. , ' 
without  pattern  scoring,  as  predictors  of  success  in  the  Basic 
Electronics  and  Electricity  School. 

It  would  be  interesting  to  continue  this  study  using  biographical 
information,  not  ordinarily  quantifiable,  instead  of  extracts  from 
current  examinations.  Biographical  questions  caiefully  constructed 
and  easily  verifiable  could  be  used  in  predicting  behavior,  and  pattern 
scoring  is  a  method  that  can  be  used  to  quantify  responses  to  these 
questions.  Responses  quantified  by  pattern  scoring,  in  fact,  will 


show  the  highest  possible  correlations  with  predicted  behavior. 
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APPENDIX  A 

The  Seven  Best  GCT  Items  Selected  by  SEQUIN 


“TO 

Form  7 
Item 
Number 

TO 

Item 

Type 

TO 

Recruit 

2  Value 

Median 
School 
p  Value 

(5) 

Median 

School 

Validity 

13 

A 

.77 

.88 

.22 

19 

SC 

.60 

CO 

• 

.20 

31 

A 

.75 

.85 

.20 

55 

A 

.41 

.49 

.24 

62 

SC 

.60 

.69 

.26 

67 

SC 

.80 

.87 

.26 

94 

SC 

.55 

.78 

.30 

NOTE:  1.  Values  in  Columns  (4)  and  (5)  are  based  on  item  data 

only  for  schools  in  which  that  item  was  selected  in  Program  SEQUIN. 


2.  A  =  analogies;  SC  =  sentence  completion  item. 


APPENDIX  B 


The  Seven  Best  ETST  Items  Selected  by  SEQUIN 


Question 

Number 

Item 

Type* 

Recruit 

P- Value 

Median  School 
P- Value 

Median 

School  Validity 

3 

M 

.57 

.71 

.34 

11 

M 

CO 

CO 

• 

.58 

.44 

13 

M 

.58 

.69 

.32 

22 

S 

.57 

.77 

,54 

40 

s 

.21 

.37 

.40 

41 

E 

.31 

.39 

.26 

50 

E 

.  25 

.31 

.28 

* 


M  =  Math;  S  =  Science;  E  =  Electricity  or  Radio 


APPENDIX  D 


Listing  of  First  Data  Preparation  Program 


THIS  PROGRAM  EDITS  RAW  DATA 


ET ST  STUDY 


INTEGER*2  CASH, ZERO, BLANK, IW,  IC3, IC4, IC5 
DIMENSION  I  W 1 8  3 } 

CATA  DASH/*-  »/,ZERO/»C  '/.BLANK/'  '/,IC3/'3  »/,IC4/» 
1  IC5/ ' 5  * / « 1 06/ *  6  '/.STAR/'-  '/,ICl/'l  */.IC2/*2  '/ 


C  CHECK  CARD  NUMBER 
10  READI4.4C0«END=50Q)  IW 
IFI IW(8 ) .EC. IC1 )  GC  TO 
IF! IWI8).EQ.1C2)  GC  TO 
I F  {  IW  {  8  )  .  EO  .  IC  3  )  GO  TO 
IF(IWCS) .EQ.IC4)  GU  TO 
IFI IWC  8) .EO. IC  5)  GC  TC 
IFI  IW<8) .E0.IC6)  GO  TO 
403  FORMAT! 80A1) 

C 

C 

C  ZEROIZE  STARS.  BLANKS.  CA! 
12  DO  20  1=1. 80 


C  ZEROIZE  STARS.  BLANKS.  CASHES 
12  DO  20  1=1.80 

IFI IWC I). EC.  STAR)  IWtl )=ZERO 
IFI  1WI I )  •  EO.  BLANK)  lift  I)=ZERO 
IFI IWC I ) .EO.DASH)  IWt I ) =ZERO 
20  CONTINUE 

WRITE! 8.303)  IW 
WRITE  16, 401)  IW 
300  FORMAT  1 80 Al) 

401  FORMAT! IX, 80A1) 

GO  TO  10 

500  IFIK.GE.2)  GO  TO  99 
K=K+1 
GO  TO  10 
99  STOP 
END 

//GO.  FTC6F001  DO  SPACE=  I CYL  »  1 5, 5 ) , P.LSc  ) 

//GO. FT 34F001  CD  UNI T=2403 , VOL=SE R=NPS416 . DI S P= IOLD, PASS ) » 

//  DC  8= l RECFM=FB . LR  ECL=6C ,3LKS IZE  =  4600) »LABEL  =  !1 »NL. .IN) , 

//  DSN=£F1 

//GO. FT34F  302  DD  UNI  T  =  2400 . VOL  =SER=NPS416 , DIS?=  I CLD. PASS ) . 

//  DC  B= I RECFH  =  FB. LR  EC  L  =  SO, BLK  S I ZE=4800 ) ,LA35L  =  ! 2.NL . .IN) . 

//  D  SN=£F2 

//G0.FTQ8F001  DO  DI SP= (NEW  .KEEP)  , UNIT =2321 , VOL =SER =CEL001 , 

//  LABEL=EXPDT  =  73180,SPACE  =  ITRK; I  57 1 ) ) . DSNAME  =  SC573. KPW2. 

//  DC3  =  (RECFM=FS.BLKSIZE=233j,LRECL  =  80  ) 
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APPENDIX  E 


SECOND  DATA  PREPARATION  PROGRAM 


READ  CARDS 

5.6 


COMPUTE  RANGE 
OF  CRITERION  SCORES 


CORRECT  ETST  QUESTIONS 
ASSIGN  .1  TO  CORRECT  RESPONSE 
0  TO  INCORRECT  RESPONSE 


BINARY  PATTERN,  PREDICTOR  SCORE, 
CRITERION  SCORE,  IN-HOUSE  I.D., 
NAVY  SERVICE  NUMBER 


RANGE  0? 
JRITERION  SCORES 


I 
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on  non  oooooo 


THIS 


APPENDIX  F 

Listing  of  Second  Data  Prepation  Program 

no  PROGRAM  EDITS  DATA  FROM  THE  ETST  TEST 
ACJ)=PREDICT0R,C(J)=CRI7ERI0N,D(J  )=SERV.NO., 


IMPLICIT  INTEGER**! A-Z ) 

DIME  NS  I  Cr.  A  ( 2500 )  ,  C  1 25  00 ) ,  D  ( 2500  J ,  W  !  7  ) 

DATA  I  SAMP » NRE  AD  tiNWRI  T E  ,  NPUNCH,  IHI  , 1  L0/2400 ,8  ,6,7  , 
156,56/ 

DO  100  Jsl'ISAMp 

IF!  J. £0.2398}  GO  TO  50 

10  READ ( NR E AD, 1,  END=CC)  8 ,  D! J  ) » KDNUM, ! W{1), 1=1,7} 

1  FORMAT  !A1,  16,1  l.Tll,UtT19,  II, T21  til  ,T30,  II, T48,  11, 
1T49, 1 1 , T58, 1 1) 

THIS  PHASE  CHECKS  FOR  THE  PRESENCE  OF  CARD  NUMBERS 
FIVE  AND  SIX 

1F(KDNUM.EQ.51  GO  TO  3 
GO  TO  10 
3  k=  1 

READ(NREAD,2 }  KDNUM , A ( J > , C ! J  ) 

2  FORMAT ( T8 , 1 i ,T52 , 12 ,T64 , 12  ) 

IF!KDNUM«EQ.6)  GO  TO  7 

GO  TO  1C 
7  K=K+1 

IF(K.NE*2)  GO  TO  10 


C 

C 

C 


THIS  PHASE  DETERMINES  RANGE 
IF(C( J) .00.0)  GO  TO  12 
IF(Cm  .LT.1L0)  IL 0=C!  . 
IF{C!J),GT«IHI)  IHI =C{ J  J 


OF  CRITERION  SCORES 


12 


IFjWjl).NF.5) 


THIS  PHASE  DETERMINES  CORRECT/ I NCQRRECT  RESPONSES 
OF  SELECTED  ET-ST  ITEMS  NOS.  3,11,13,22,40,41,  AND 

W  (  1 J  =  0 
W  1 2  I =0 
W  (  3 )  =  0 
W ( 4 )  =  0 
W  {  5  )  =  0 
W  (  6 )  =  C 
W  (  7  )  =  0 


50 


C 

C 

c 


20 


IF(W(2).NE.5J 
IF! W(3) .NE.5) 
IF(W(4).NE.2) 
IF  C W( 5> .NE.3) 
IF!W!6) «NE .2 ) 
IF IW (7 ) .Nc  .3  ) 
00  20  1=1,7 
IF(WCI).EO.O) 
W(I)=l 
CONTINUE 


GO  TO  20 


CELL,  PAPER 


OUTPUT  INFO  ONTO  CARDS,  DATA 
»J»  IS  USED  AS  INHOUSE  ID 

WRITE (N WRITE, 30)  ( W l I )  ,  I  =  1 , 7 ) , C  !  J )  ,  A { J)  , J , B , D( J ) 
30  FORMAT (4 ( IX i7 1 1,12, 12,  14 ,7 20 , A 1 , 1 6, 5X } ) 

WRITE! N PUNCH ,4  0)  ( W (I)  , I =1 , 7)  ,C  !  J ) , A{ J ) , J , E , D CJ ) 
WR IT5 ( 4, 40 )  !W( I), 1=1, 7),C(J)»A1J1 , J ,3 ,D ( J ) 


40 

10; 

50 

25 


I2»I2,I4,T20,A1,I6) 


FORMAT (71 l, 

CONTINUE 

WRITEC6,25)  IHI,ILO 
FORMAT!'  IHI  EQUALS' 

STOP 
END 

//GO. FT06FQ01  DD  SPACE=!C' 

//G0.FTC7F  C01  DD  SYSOUT=B 
//GO. FT08F001  DD  DSN=S0575. KPW2,UN IT= 2321, VOL= SER=CEL 001 , 

//  DC8  =  (  RECFM=F3 .8  LKSI  ZE=20  JO ,  LRECL=30 ) ,  DIS  P=  ( CL C, KES-P  ) 

/VG0.FTG4F001  DD  DISP=(NEW,KCE?) ,UNI T=2321 ,VOLUME=SER=CSL001 
//  L ABEL  =  EXPDT =73180, SPACE=(TRK, (571)), DSNAME=  S0575.KPW3, 

//  CC8=(RcCFM=F8,BLKSI ZE=2000, LRECL=80) 


*12,/'  ILO  EQUALS’, 12,/) 


!5 ,5 ) ,RLSE) 
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c 

c 

c 

c 

c 

c 

c 

c 

c 

c 


c 

c 

c 

c 


c 

c 

c 

c 

c 


c 

c 

c 

c 


APPENDIX  G 

Listing  of  Validation  Program 

THIS  PROGRAM  WORKS  ON  ODD  NUMBERED  QUESTIONS  FROM  THE 
ETST  EXAM.  THE  MULTIPLE  CORRELATION  COEFFICIENT  IS 
CALCULATED  AS  WELL  AS  THE  CORRELATION  COEFFICIENT 
BETWEEN  PATTERN  AND  T  OTAL  CORRECT. 

REAL^IPci «  C2.A1 » A2?V,  W  ,  X,R1  »R2»  R3,R4,R5,Q,Z1,Z2»Z3,Z, 

1CIMENSION*A?1200),B{12  00),C{ f 200 ) ,D{ 1200) ,E H 200)  , 

IF ( 128 ,47).G(123).H(128 ) , P (7  *1200 )  ,S  { 128 )  ,X  ( 1200) 

DATA  Nl,N2,N3,N4/l 182 ,7,47.128/ 

L=0 

READ  IN  DATA 

READ?9» 9 ) IP(I,J),I=1*N2)»C(J) ,AC J) »D( J) 

FORMAT  1711,12,12,14) 

IOUM  IS  A  DUMMY  VARIABLE  CONTROLLING  THE  READING  OF 
EITHER  EVEN  OR  ODD  DATA. 

READ19.2)  1DUM 
2  FORMAT  (ID 
13  CONTINUE 

COUNT  TOTAL  ONES  FOR  EACH  SUBJECT 

DO  15  J  =  1 « N1 
E  ( J )  =  0 
DO  12  1=1, N2 
EIJ)=P(I  ,J)+E( J) 

12  CONTINUE 
15  CONTINUE 
I  Al  =  3 
I  A2=0 
IV=0 

DO  8  J=1,N1 

IA1,  IA2,  IV  ARE  VARIABLES  TO  BE  USED  IN  THE  CALCU¬ 
LATION  OF  R (TOTAL  ONES)  LATER  IN  THE  PROGRAM. 

I A1=E ( J ) +1 A1 
IA2=E(J)*£{J)*IA2 
■  IV=C( J ) *  E( J ) +1 V 
CONTINUE 

OUTPUT  VALUES  FOR  IAI,  IA?  TO  BE  USED  IN  THE 
COMPUTATION  CF  TEST-RETEST  CORRELATION  COEFFICIENT. 

WRITE (6,999)  1A1,IA2  ,  „  .  ,  .  „  ,  T  _ 

999  FORMAT (T20, *  I A 1= ' , I  8, // , T20, f IA2  =  ' ,  I  8) 

DETERMINE  THE  JOINT  FREQUENCY  DISTRIBUTION  OF  PATTERN 
AND  CRITERION  SCORES  (THE  SECOND  I  LOOP  CCNVcRTS 
BINARY  NUMBER  PATTERNS  TO  DECIMAL  EQUIVALENTS 
TO  SERVE  AS  ROW  ADDRESSES. 

DP.  14  T=).N'4 
DO  17  J  =  1 , N  3 
F  C I , J )  =  0 


C 

r 

■v* 

C 

C 

c 


8 


C 

C 

c 

c 

C 


c 

C 

C 

C 

c 

c 

c 
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-  ”  - ■».  j - ^=--rr^a»=-^-v^--^-s.-y-  r— y-  =-*--  it-  -c  - -  — 


17 

14 


19 


C 

C 

C 

c 


18 


21 


10 

20 


C 

C 

C 

C 


25 


31 


C 

C 

C 

C 

C 
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C 

C 

C 


Listing  of  Validation  Program 
(Continued) 


CCNTINUE 
CONT INUE 
CO  18  J  =  l«  Ni 
M=1 
K=N4 

DO  19  I  =  1«  N2 
K=K/2 

M=K*Pl I . J)+M 
CONT  INUE 
N=C{ J)-25 
F!M#N )=F!M<N}+1 
B(J)=M 
CONTINUE 


CCMf’UTATION  OF  PATTERN  SCORES 


CO  20  1=1. N4 

S1=0 

S2=0 

DO  21  J  =  1.N3 
S2=F (I . J) +S2 
S1=(J+29)*F{I.  JH-S1 
CONTINUE 

IF(S2.FQ.O)  GO  TO  10 
S(I5=Sl/$2 
GC  TO  20 
SC  I )  =- 1 

CONTINUE  '  4  .  .  . 

WRITE (7 .25 ) (S( 1 ) tF=1.N4) 
FORMAT! I OF  7- 4) 


ASSIGNMENT  OF  PATTERN  SCORES  TO  SUBJECTS 


CO  31  J  =  1 » N l 
K=B ( J ) 

X(  J  )  =  SC  K  ) 
CCNTINUE 


COMPUTATION  OF  CORRELATIONS 


A1=0.D0 

A2=0.D0 

C1=0.D0 

C2=0. DO 

X2=0.00 

X1=0.D0 

V=0.00 

W=0«  DO 

L’U=0. 

00  41  J=1.N1 
C1=C( Jl+Cl 
C2=C l  J  )  *C  ( J  )  +C2 
A1.=A  !  J  J  +A1 
A 2= A (J)*A(J)+A2 
X1=X( J )+Xl 
X2=X( J)*X! J)+X2 
V=C! J )*Al J )+V 
W=C(  J)?X(J  >+W 
UU=E( J)*X( J)+UU 
CONT  INUE 

R3=(N1*C2)-(C1*C1) 
R2=(  N15?X2J  — (  XI*  XI) 
R5=(N1*W )— (Cl-^Xi ) 
0={R2*R3}**3.5 
R2=R5/0 


FIRST  TIME  THROUGH  PROGRAM  R1  IS  THE  CORRELATION 
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t4 


Listing  of  Validation  Program 
(Continued) 

rnFFFlfTFNT  FOR  PATTERN*  SCORE  VS.  CRITERION.  SECOND 
T?HE  THROUGH  R1  IS  CORRELATION  COEFFICIENT  FOR  TOTAL 
ONES  VS.  CRITERION. 

R1  =  {NI*A2)~(A1*A1) 

R4= I Nl^V )— ( C I^A 1 ) 

C=IR1*R3  )**0.5 
Rl=R4/0 

IFtL.EO.l)  R1STAR=R1 

COMPUTATION  OF  TEST  STATISTIC  FOR  R  DIFFERENCE 

Z1  =  {1+R1)/(I'-R1) 

Zl=DLOG(Zl)/2 
IF(l.EO.l)  GO  TO  40 
Z2=(1+R2)/(1~R2) 

Z2=DLOG(Z2)/2 
Z3=2 ./ ( NI-3 . ) 

Z3=IZ3>**0.5 
Z=(Z2-Z1)/Z3 
IF  (L.EO.l)  GO  TO  90 

CONSTRUCTION  OF  RESPONSE  PATTERNS 

G( 1 )=2 
H(1 1=0 
G( 2) =1 
H( 2 )  =  1 
K=1 
N=1 
N=2*N 

IF(N.GE.N4)  GO  TO  60 

K=10*K 

DO  51  1  =  1.  N 

G( N+I ) =G  1 1 ) +K 

H(N+I)=H(I 1  +  1 

G( I )=Gt I }+2*K 

CONTINUE 

GO  TO  50 


ORDERING  OF  RESPONSE  PATTERNS 

N=N4— 1 
K=0 

CO  80  1=1, N 
V=G( I J 

IFH(s!l+l).GE.SCI)}  GO  TO  80 
U=S(I } 

V=G( I ) 

W=H(  I ) 

sm=sti+u 

Gt  n=G(  i +i  > 

H(  I ) =H ( I  +  1 ) 

SU*1)=U 
G( 1  + 1 J  = V 
H(I+1I=W 
K  =  1 

CONTINUE 

IF  (K.EO.l)  GO  TO  70 


PRINT  OUT 


NUM=1 

WRITlc6,1001(G(I),H{I),SII),I=NUM,NyMl) 

"FORMA”  !*rl*  ,61/1  ,760,‘EiSt  cXAM*  ,21/ )  ,T45  ,*  +  * 
8j8( *-*  i , *  +  * T45, 1  i*, ix, ‘Pattern*  , 
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Listing  of  Validation  Program 
(Continaed) 


THE  VARIABLE  R7  IS  THE  CORRELATION  COEFFICIENT  BETWEEN 
PATTERN  SCORES  ANO  TCTAL  CORRECT  (ONES). 

R7=N1*UU-I A1*X1 

WRI TE ( 6,33  5  R7,UU 

FORMAT ( •  R  ?  =  *  »  F  i  8 . 4 , *UU=* ,F18.4) 

FORMAT  ( c  R7  =  '  ,  *=1 8. 4  ,  *UU  =  * ,  F18 .4  ) 

R1=IN1*A2)-CAI±a1) 

R8=(N1*X2)-{X1*X1) 

0=  { R1^R8  )*'*  .  5 
R7=R7/0 

WRITE (6 « 21 3 )  R 7 

FORMAT (« 0* ,»R{ PATTERN  SCORE/TOTaL  ONES)  EQUALS *, F6. 3} 

RMIJL  IS  THE  MULTIPLE  CORRELATION  COEFFICIENT  TO  BE 
USED  IN  THE  DETERMINATION  OF  'F*. 

RMUl=( ( R2**2)  +  t  R1 ST AB**2 )-2* ( R2*R1ST  AR*R7 )  )/ ( l-R7**2 ) 
RHUt=RMUL**.5 

FF= ( ( RMUL**2 )- ( R1ST  aR**2 ; )=?( Nl-3)/ \ 1~ I RMUL**2) ) 

WRITE ( 6*41 7}  RMUL,FF 

FORMAT ( *0*  » *RI MULT •  CCRRE L.  C0EF.5  EQUALS* ,4X,F6. 4, //, 
1*  FF  EQUALS*  ,F8.4) 

THIS  PORTION  OF  THE  PROGRAM  IS  USED  IN  THE  DETERMINA¬ 
TION  OF  A  FREQUENCY  0 1  STRI  BUT  I  CNF  I.g.  PATTERN 
VS.  NUMBER  OF  PEOPLE  WITH  THAT  PATTERN 


//GO 

//GO 

//GO, 

// 


DO  350  1=1,128 

WRITE ( 6,357)  GULSU) 

FORMAT i T20, ' A  PATTERN  OF : * ,2X , 1 7 , 2X , *  AND  PATTERN  SCORE 
13X, F8*4, // ) 

KOUNT^O 
DO  355  J=1,N1 

IFCS(l).EO.XtJ))  GO  TO  359 
GO  TO  355 
K0UNT=K0UNT+1 
WRITE ( 6 , 352 )  OtJ) 

FORMAT  tT15,I4) 

CONTINUE 

WRITEI6 , 351)  KOUNT 

FORMAT ( *  TOTAL  PEOPLE  HAVING  THIS  SCORE: * , 1 3* // ) 

CONTINUE 

STOP 

END 

» FTC6F00.1  DD  SPACE = ( C YL » l 5 ♦ 1 ) )  ,SYSCUT=D 
. FT 07FO0 1  DD  SY$0UT=8 

, FTC9F001  DD  DSN=S057 5 .KPW3 , UNIT  =  232 1 , VOL=S ER=CSLOO 1, 
OCB=( RECFM=FB,B  LKSI Z E=2 000, LRECL  =  SQ),DISP={CLD, KEEP) 
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APPENDIX  H 

Output  I  -  Pattern  Information 


ETST  EXAM 


PATTERN  {  TOTAL  ONES  I  PATTERN  SCORE 


2222112 

2 

-1.0000 

2222111 

3 

-1.0000 

2212121 

3 

-1.0000 

2212111 

4 

-1.0C00 

2122121 

2122112 

2122111 

2121122 


2112211 

2112112 

1212111 

1122121 


1122112 

1122111 

2212211 

2222211 


“1.0000 

-1.0000 

-1.0000 

-1.0000 


-1.0000 

-1.0000 

-1.0000 

-1.0000 


-1.0C00 

-1.0001 

48.5000 

49.0000 


2222222 

Q 

i  49.6071 

1222222 

1 

51.3182 

2222121 

2 

52.0000 

2122211 

3 

52,0000 

2221222 

1 

52.5238 

2222212 

1 

52.5833 

1222121 

3 

53.5000 

2222221 

1 

53.6667 

2122222 

1 

53.8CQ0 

2121211 

4 

54.0000 

2222122 

1 

54.1667 

2221212 

2 

54.3CC0 

2212122 

2 

54.50C0 

2212222 

1 

54.5500 

1212222 

2 

54.7500 

1222212 

2 

54.8889 

NOTE  i:  IN  PATTERNS*  2’SREPRESENT  0* S 
NOTE  2:  A  PATTERN  SCCRE  OF  -1  INCICATES  A  PATTERN  NO  ONE  HAS 
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APPENDIX  H 


NOTE  2: 


Output  I  -  Pattern  Information 


ETST  EXAM 


PATTERN 

TOTAL  ONES 

PATTERN  SCORE 

2221221 

2 

56,0000 

2112111 

5- 

56,0000 

1222122 

2 

26.3333 

2122221 

2 

56.2000 

2122212 

2 

56.5000 

1222112 

3 

56.2C00 

1212122 

3 

56.8000 

2121212 

3 

56.8750 

2212212 

2 

57.2500 

1122222 

2 

57.3333 

1221222 

2 

57,3913. 

112222  . 

3 

57.6000 

2211222 

2 

57.7308 

2221121 

3 

58.0000 

2121222 

2 

58.0000 

2112212 

3 

58.0000 

2211212 

3 

58.2941  { 

1212212 

3 

58.3077 

122 1221 

3 

58.3333 

2221211 

3 

58.5000 

2211122 

3 

58.7143 

1212221 

3 

58.8750 

2211221 

3 

58.9091 

2221122 

2 

59.0000 

-j * 

2211211 

4 

n  . 

59.0000 

1222111 

4 

59.0000 

1221212 

3 

59.2222 

1221211 

4 

59.2500  j 

1222221 

2 

59.5000  1 

2121112 

4 

59.6667 

2112222 

2 

59.7143  j 

1211222 

3 

60.0682  { 

NOTE  l: 

IN  PATTERNS*  2*  SREPRESENT  0*S 

A  PATTERN  SCORE  OF  -1  INDICATES  A  PATTERN  NO  ONE  HAS 


(Continued) 
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APPENDIX  H 

Output  I  -  Pattern  Information 

ETST  EXAM 


+ — 


PATTERN  I  TOTAL  ONES  I  PATTERN  SCORE 


{  ctliZZZ 

3 

60*7333 

1221122 

3  1 

60.7500 

I  2121221 

3 

61.0000 

j  3.12X222 

3 

61.0000 

f  U12122 

4 

61.2500 

I  2212221 

2 

61.6667 

I  1212211 

A 

61.6667 

|  2221112 

3 

62.0000 

1  22.12112 

3 

62.0000 

2122122 

2 

62.0003 

1  2111212 

A 

62.2222 

i  1 21X221 

A 

62.2308 

j  21X2121 

£ 

62.5000 

t  1212112 

4 

62.5000 

1122212 

3 

62.5714 

*1 12222 

3 

62.8621 

2112221 

3 

63. COCO 

1112232 

A 

63.2609 

1211212 

A 

63.2857 

1122122 

3 

63.5000 

1121212 

4 

63.5625 

1211X22 

4 

63.6000 

1112121 

5 

63.7500 

2121121 

4 

64.0000 

2112122 

3 

64.0000 

1211211 

5 

64.0000 

1 1222X1 

!  4 

64.2500 

1111222 

A 

64.4000 

2111122 

4 

64.6GC0 

1121122 

4 

64.8000 

2211121 

4 

65,0000 

2111221 

4 

65.0000 

NGTE  2:  A 


NOTE  is  IN  PATTERNS,  2*  SREPRESENT  0* S 
PATTERN  SCORE  CF  -1  INDICATES  A  PATTERN 


NO  ONE  HAS 


i-m 


I  m 


-K 
‘  =1 


SI 
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APPENDIX  H 


NOTE  2: 


Output  I  -  Pattern  Information 


ETST  EXAM 


PATTERN 

TCTAL  ONES 

PATTERN  SCORE 

2211112 

A 

65.2000 

1221112 

4 

65.2000 

1211112 

5 

65.3000 

1121221 

4 

65.3000 

1112211 

5 

65.5000 

1111212 

5 

65.6C97 

1112112 

5 

65.7500 

2111121 

5 

65.7778 

1112221 

4 

65.8571 

2111112 

5 

66.0000 

1222211 

3 

66.0003 

1121112 

5 

66.3333 

2121111 

5 

66.5000 

2111211 

5 

66.7500 

1111221 

5 

66.7667 

1111122 

5 

67.7273 

2221111 

4 

68 .0000 

1221111 

5 

68.3333 

1211121 

5 

68.5000 

1121211 

5 

68.5000 

1111211 

6 

68.5000 

1111112 

6 

68.7500 

1212121 

4 

69.5000 

1121121 

5 

69.6667 

2211111 

5 

69.8333 

2111111 

6 

70.0C00 

1112111 

6 

70.5300 

1211111 

6 

70.8000 

1221121 

4 

71 .0000 

1111121 

6 

71.2954 

1111111 

7 

73.2857 

1121111 

6 

73.6667 

NOTE  is  IN  PATTERNS*  2  * SRePRESENT  0'S 
A  PATTERN  SCORE  OF  -1  INDICATES  A  PATTERN  NO  ONE  HAS 
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APPENDIX  I 


Output  11  -  Subject  Information 

ETST  EXAM 


IDENT  i  PREDICTOR  1 

1  CRITERION 

i  PATTERN  SCCRE 

1  TOTAL  ONES 

1 

66 

76 

63.7500 

6 

3 

59 

53 

54.7500 

2 

t; 

57 

54 

54.7500 

2 

7 

73 

73 

71.2954 

6 

9 

l  56 

60 

58.9091 

3 

11 

71 

70 

66.0000 

5 

13 

63 

64 

64.4000 

4 

15 

59 

57 

54.5500 

1 

17 

;  64 

56 

57.3913 

2 

19 

1  64 

54 

49.6071 

0 

21 

1  46 

58 

54.7500 

2 

23 

|  56 

56 

58.8750 

3 

25 

56 

60 

65.6097 

5 

27 

1  55 

54 

58.2941 

3 

29 

1  b-j 

68 

i  68.7500 

6 

31 

49 

57 

54.7500 

2 

33 

55 

62 

60.0682 

3 

35 

72 

76 

73.2857 

7 

37 

68 

71 

6S.7500 

1  6 

39 

66 

68 

65. 8571 

1  4 

41 

73 

71 

70.0000 

6 

43 

65  1 

60 

64.4000 

4 

45 

59 

59 

61.0000 

3 

47 

63  1 

76 

73.6667 

6 

49 

65  ! 

64 

62.5000 

4 

51 

54  1 

38  1 

51. *3 18  2  I 

1 

53 

66 

64 

62.8621  i 

3 

55 

58 

62 

57.25GC 

2 

57 

51 

59 

57.2500  | 

2 

59  ! 

61 

64 

65.2000  1 

4 

61  ! 

62 

68 

65.3000  I 

4 

63  1 

63  1 

67 

7C.0000 

6 

i  65  ! 

60 

62 

56.0000 

2 

67  I 

66 

70 

68.5000 

5 

69  I 

1 

68  1 

69 

63.6000 

4 

71 

6Q 

68 

68.5000 

6 

73 

62  •  I 

60 

54.5500  I 

1 

75 

58 

60 

56.8750  ! 

3 

77 

65  1 

59 

54.7500 

2 

79 

58 

58 

61.0000  1 

3 

81 

62 

57  i 

56.0000 

2 

83 

63 

71 

69.5000 

4 

85 

59  ! 

58  I 

57.7308 

2 

87 

70 

76 

73.2857 

7 

89 

59 

70 

65.7778 

5 

91  ! 

51 

61  ! 

60. 0682 

3 

93 

51 

49 

52.5238  1 

1 

95  i 

45 

60 

57.6000 

3 

97  i 

66  1 

68 

68.5000 

5 

99 

71 

65 

68.7500  | 

6 

56 


C  APPENDIX  J 

C  Cross-Validation  Listing 

C  CROSS  VALIDATION 

C 

INTEGER* A  A,C»  D,  E,G,H,  P 

REAL*8  C1,C2,A1,A2,V,W,X,R1,R2,R3,R4,R5,C,Z1,Z2.Z3,Z, 
1 AA,BB 

DIMENSION  A{1200}»B(12Q0) ,C( 1200) ,0(1200) , E(1200), 
1G(123)«H(128)«P(7, 1200 )  ,  S(  128) ,X{ 1200) 

C 

C  NOTE:  PARAMETERS  OF  DATA  CARD 

c 

DATA  Ml  * N2 * N3« N4/1182, 7,47,128/ 

L-0 

C 

C 

C  READ  IN  DATA 

C 

DO  13  J=1.N1 
C 

c 

C  THIS  CROSS  VALIDATION  PROGRAM  READS  DATA  FRCM  THE 

C  ET ST  EXAM 

C 

c 


C  IDUM  IS  A  DUMMY  VARIABLE  CONTROLLING  THE  READING  OF 

C  EITHER  EVEN  OR  OLD  CAT  A. 

C 

READ(4,2)  IDUM 

2  FORMAT  (ID 

READ (4 ,9)  (PM  ,J),I=1,N2),C(J)  ,A(J),D(J) 

9  FORMAT (71 1,12, 12, 14) 

13  CONTINUE 

C 

C 

C  READ  REGRESSED  MEAN  AND  WEIGHT  COMPUTED  FRGM  THE 

C  MAIN  PROGRAM. 

C 

READ! 5,31  AA,B  8 

3  FORMAT (  F6 . 3  «  F6  .3  ) 

C 

C 

C  READ  PATTERN  SCORES  CALCULATED  FROM  MAIN  PROGRAM 

C 

READt  5*  10)  (SU  )  ,1  =  1, N4) 

10  FORMAT ( 10F7 .4 ) 

C 

c 

C  COUNT  TOTAL  CORRECT  FOR  EACH  SUBJECT 
C 

DO  15  J=1,N1 
E( JJ=0 

DC  12  I = 1 »  N2 
E( J)=P( I ,J)+E( J) 

12  CONTINUE 

15  CONTINUE 

IA  1=0 
IA2=0 
IV=0 
C 
C 

C  IA1,  I A2,  IV  ARE  VARIABLE  TO  BE  USED  IN  THE  CALCU- 

C  LATION  OF  R( TOTAL  CNES)  LATER  IN  THE  PROGRAM. 

C 

CO  8  J=1 »NI 
IA1=E( Jl+IAi 
IA2=E(J,'*?(J)  +  IA2 
1 V=CC J )*  E( J ) +1 V 
8  CONTINUE 
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Cross-Validation  Listing 
(Continued) 


OUTPUT  VALUES  FOR  IA1,  AND  IA2  TO  BE  USED  IN  THE 
COMPUTATION  OF  TEST-RE  TEST  CORRELATION  COEFFICIENT. 

WRITE! 6,999)  1AI,IA2 

FORMAT (T20, * 1A1=' , I 8, //, T20,  *  I A2=* ,18) 

CALCULATE  DECIMAL  ECUI VALENT  CF  BINARY  PATTERNS 
THE  8(J)  ARRAY  HOLDS  DECIMAL  bOUl VALENT  OF  EACH 
SU3JECT  *  S  BINARY  PATTERN. 


CO  18  J  =  l,  N1 

M=1 

K=N4 

DO  19  I  =1 ,N2 
K=K/2 

M=K*P( 1 »J)+M 
CONT  INUE 
B( J)=M 
CONTINUE 

ASSIGNMENT  OF 
DO  31  J=1.N1 
K=B(J) 

XC J)=SfK) 
IF(XIJ)  .LT.O) 
CC  NT  IMUE 


PATTERN  SCORES  TO  SUBJECTS 


X(  Jt)=EtJ)*BB+AA 


COMPUTATION  OF  CORRELATIONS 

Al=O.DO 

A2=U. DO 

Cl=O.DO 

Xl=O.DO 

X2=0. DO 

W=O.DO 

V=0. DO 

L‘U=0. 

C2=0«  DO 
CO  41  J  =  1,N1 
Cl=C(J)+Cl 
C2=C(J)*C(J)+C2 
A1=ACJ)+A1 
A2=A(J)*A( J)+A2 
X1=X( JJ+Xl 
X2=XC  J)*^Xl  J)+X2 
V=CIJ)*A( J)+V 
W=C(J)*X(J)+W 
UU=E  {J)"*X(J)+UU 
CONTINUE 

R3=tNl*C2)-(Cl*Cl) 
R2=CN1*'X2I  — ( X1*X1 ) 

R  5=  ( N 1*  W )-  ( C 1“  X 1 ) 

Q-MR2*R3><*0.5 

R2=R5/0 


FIRST  TIME  THROUGH  PROGRAM  R1  IS  THE  CORRELATION 
COEFFICIENT  FOR  PATTERN  SCORE  VS.  CRITERION.  SECOND 
TIME  THROUGH  R1  IS  CORRELATION  COEFFICIENT  FOR  TOTAL 
CORRECT  VS.  CRITERION. 

R1=(NI*A21-(A1*A1) 

R4=(N1*V)-(C1*A1) 

0=(Rl*R3)**0.5 

Rl=R4/0 

IFIL.EO.il  R1STAR=R1 


COMPUTATION  OF  TEST  STATISTIC  FOR  R  DIFFERENCE 
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Cross-Validation  Listing 
(Continued) 


Z1=C1+R1  )/  tl-Rl ) 
7.1=0100(7.1  )/2 
IF(L.EQ.l)  GO  TO  * 
Z2=(l+R2)/ (1-R2) 
Z2=DL0G(Z21/2 
Z3=2./ (  N 1— 3  »  ) 

Z3=( Z3 )**Q»5 
Z=(Z2-Z1)/Z3 
IF  (L.EO.l)  GO  TO 


PRINT  OUT 

NUM2=1 

NUM3=50 


nun 

WRITE(6,200HD( J),  A(J  } . C ( J ) , X( J ) , E ( J ) , J=NUM2, NUM3) 
FORMAT!’)'  ,4(/) .T60,'ETST*  ,T67  ,  *  EXAM*  t  2  ( /  )  *T34,  •  +  •  , 
60(’-’ )  ♦  '+• , /, T34 , *  1  ' « IX. ' IDE NT* ,  IX, •  I '  ,lXt 
•  PREDICTOR* , IX.’ I ’, IX* 'CRITERION'* IX,*  I  • , 


160( ’— * ),'+',/. T 
2*  PREDICTOR’ , IX 
3*  PATTERN  SCORE 
A*  I  '  *T95*  *  I  *  »/ * 
52X«* I  *  ,T49.12,T 
6T82. *  I  ‘ , T89.il, 
7T35*60 ( *— • ),/} ) 


THE  NEXT  TWO  IF  STATEMENTS  CONTROL  THE  NUMBER 
AND  LENGTH  OF  THE  LAST  TABLE 


THE  FIRST  ‘IF*  STATEMENT:  NUMBER  INSIDE  PAREN 
SHOULD  BE  ONE  MULTIPLE  OF  ’5’  HIGHER  THAN  Nl;  E.G.  IF 
N1 =627 »  NUMBER  INSIDE  PAREN  SHOULD  BE  630;  IF  Nl=986, 
NUMBER  INSIDE  PAREN  SHOULD  BE  990. 


IFINUM3.E0.1190)  GO  TO  221 
NUM2=NUM2+  50 
NUM3=NUM3+50 


SECOND  'IF*  STATEMENT:  NUMBER  INSIDE  t>AP.EN  MUST  BE 
CNE  MULTIPLE  OF  50  +1  LOWER  THAN  Nl;  E.G.,  IF  Nl=1286, 

till  (Hoc  r\  t  Mf  t  nf!  r*  a  n  r  N  nmin  n  nr  i  c  i  .  t  r  _  i 


CNE  MULTIPLE  OF  50  +1  LOWER  THAN  Nl 
NUMBER  INSIDE  PAREN  SHOULD  BE  1251; 
NUMBER  INSIDE  PAREN  SHOULD  BE  101. 


IF  Nl= 126, 


IFCNUM3.LT. 1151)  GC  TC  199 
NUM2= 1151 
NUM3=1 190 
GO  TO  199 
221  A1=IA1 
A2=IA2 
V=IV 

209  WRITE ( 6. 210)  R2.R1.Z 
210  FORMAT!* 1* , ‘CORRELATION  AND  TESTS'//*  ** 

1*  R( PATTERN)  EOUALS *  ,F 15. 7,/ , 

2  'R (PREDICTOR )  EOUALS’ , F15. 7, /, 

3*  Z  EOUALS* • FI  5.7/ ) 

L=1 

GO  TCi  30 

90  WRITEI6.212)  Rl.Z 

212  FORMAT (’ 0* » 'R(  TOTAL  CORRECT)  EOUALS* ,F15. 7/ 
1*  Z  EQUALS* ,F15. 7/ ) 


THE  VARIABLE  R7  IS  THE  CORRELATION  COEFFICIENT 
BETWEEN  PATTERN  SCORES  AND  TCTAL  CORRECT  (ONES) 

R7=N1*UU-IAI*X1 

WRITE (6. 37)  R7.UU 

FORMAT!*  R7  =  ‘  ,F1S.  A,*UU=*  ,F1S.4>* 

R1=CN1*A2K(A1*A1) 
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Cross-Validation  Listing 
(Continued) 


R8={ N1*X2) — (Xl*Xl ) 

0={R1*R8)**.5 

R7=R7/Q 

WRITE(6*213)  R7 

213  FORMAT ( *0' * *R l PATTERN  SCORE/TOTAL  ONES)  EQUALS* ,F6. 3) 


RMUL  IS  THE  MULTIPLE  CORRELATION  COEFFICIENT 
TO  BE  USED  IN  THE  DETERMINATION  OF  *F*. 


RMUL  =  (  (R2**2)4-(R1$TAR**2}-2*{R2*R1STAR*R7)  )/U-R7**2) 
RMUL=PMUL*-*.5 

FF=URMUL**2i-{RlSTAR**2) )* (Nl-3)/ {1-{RMUL**2 ) ) 
WRITE(6*417)  RMUL*  FF 
417  FORMAT C » 0*  **RC MULT.  CCRREL.  COEF.)  EQUALS  * *4X, F6. 4, // 
1*  FF  EQUALS*  *F8.4) 

STOP 

END 

//GO.FTC6F001  DD  SPACE=(CYL* (5*1) ) 

/ /G0.FTQ7F001  DD  SYSOUT=B 

//GG.FT04F001  OD  0SN=S0575.KPW3-,UNIT=2321  »V0L=SER=CEL001* 

7/  DCfi=(RECFM=F8,BLKST2E=2000»LRECL=80) ,DI SP=(CLD,KEEP) 
//GC.SYSIN  DD  * 


APPENDIX  K 


Output  III  -  Cross-Validation 

(First  50  Subjects) 

ETST  exam 


-is? 


I  DENT  1 

1  PREDICTOR  1 

I  CRITERION 

1  PATTERN  SCORE 

I  TOTAL  ONES 

2  1 

58  1 

62  1 

51.3182 

1 

4  ! 

72 

76 

71.295A 

6 

6 

65 

67 

58.  5000 

3 

a 

59  1 

52  1 

52.5238 

1 

10 

62  ! 

61 

62.2308 

A 

12 

71 

76 

1  73.2857 

7 

14 

66 

6?  ; 

1  6A.A000 

A 

16 

63 

62  1 

1  65.3000 

5 

18 

A8 

36 

1  6A.2500 

A 

20 

50 

52  1 

[  52.5833 

1 

22 

63  1 

66 

66.3333 

5 

24 

61 

53 

5A.5530 

1 

26 

50 

56 

57.7308 

2 

28 

59 

65 

5A.7500 

2 

30 

62  1 

1  6A 

64.A000 

A 

32 

61 

58 

54,7500 

2 

3A 

71 

68 

65.6097 

5 

36 

65 

62 

62. 8621 

3 

38 

57 

58 

59.71A3 

2 

AO 

66 

62 

57.7308 

2 

A2 

65 

58 

65.8571  1 

A 

AA 

61 

56 

52.5833 

1 

A6 

68 

66 

69.6667 

5 

A6 

70  ! 

75 

73.2857 

7 

50 

67  I 

59 

52.5238 

l 

52  I 

62  1 

61 

64*8000 

A 

5A 

68  1 

68 

65.6097  1 

5 

56 

66  ' 

65 

61.0000  i 

3 

58  1 

61  ! 

66 

71.295A  1 

6 

60  1 

59  1 

5A 

60.7333  i 

3 

62 

54  ! 

55  1 

62.8621 

3 

6A 

55 

52 

62.8621 

3 

66 

51  I 

58 

60.7333 

3 

68 

56  i 

51 

49.6071 

0 

70 

62  l 

57  1 

54.3000 

2 

72  1 

60  j 

61 

62.2308  ! 

A 

7  A  1 

73  1 

67  | 

68.75GG 

6 

76  ! 

65  ’  J 

7 A  1 

73.2857 

7 

78  i 

52  : 

51 

49.0000 

2 

80  I 

63  j 

7A  I 

66.7667 

5 

82 

66  | 

63 

54.0C0C 

A 

8A 

70 

72 

67.7273 

5 

86 

63 

6A 

64. 4000  ! 

A 

88 

55  1 

51 

5A.55C0  1 

1 

90 

67  1 

69 

67.7273  1 

5 

92  ; 

70 

71  1 

68.7500  ] 

6 

9A 

67 

76 

67.7273  1 

5 

96  1 

•  53 

6A 

65.0000 

A 

98  1 

60 

62 

57.3913 

2 

100  1 

72 

62 

67.7273 

5 

APPENDIX  L 

GLOSSARY  OF 
COMPUTER  VARIABLES 
USED  IN  THESE  PROGRAMS 


A(j)  -  jth  subject's  GOT  score  used  as  a  predictor 

A1  -  sum  of  predictors 

A2  -  sum  of  squares  of  predictors 

B(J)  -  jth  subject's  decimal  value  of  his  binary  score 

C(J)  -  jjh  subject’s  final  school  grade  used  as  the  criterion 

Cl  -  sum  of  the  criterion  scores 

C2  -  sum  of  squares  of  criterion  score 

D(J)  -  jth  subject's  identification  number 

E(j)  -  jth^  subject’s  total  correct 

F(,)  -  joint  frequency  distribution 

G( )  binary  pattern  (2  replaced  O  in  output) 

H(  )  total  correct  in  a  binary  pattern 
1A1  -  sum  of  total  correct  (total  ones) 

1A2  -  sum  of  square  of  total  correct  (total  ones) 
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used  in  calculating  range  of  criterion  scores 


IV  -  sum  of  C(J)*E(J) 

IW( )-  a  column  on  a  subject's  record  card 
KDNUM  -  card  number 
M  -  column  in  'F  matrix* 
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(Continued) 

N  -  row  in  ‘F  matrix* 

N1  -  sample  size 

N2  -  number  of  elements  in  the  binary  pattern 
N3  -  range  of  criterion  scores 

N4  -  2**N2;  number  of  combinations  of  patterns  of  1/0  with  N2 
questions 

P(, )  -  jth  subjects  pattern  of  ones/zeros 

R1  -  correlation  coefficient  between  criterion/predictor 

2nd  time  corelation  coefficient  between  criterion/total  correct 

R2  -  correlation  coefficient  between  criterion/pattern 

R3-5-  correlation  coefficients  used  in  determining  R1  and  R2 

R7  -  correlation  coefficients  between  pattern  scores  and  total  ones 

RMUL  -  multiple  correlation  coefficient 

S(I)  -  a  pattern  score  associated  with  a  particular  pattern 

51  -  weighted  sum  of  people  with  that  pattern  (weights  being  the 

criterion  scores) 

52  -  number  of  people  with  that  pattern 

V  -  sum  of  product  of  C(J)*A(J) 

W  -  sum  of  product  of  C(J)*X(J) 

W(l-7)  -  an  array  of  questions  being  used  in  this  study 
X(j)  -  jjh^  subject’s  pattern  score 
Xi  -  sum  of  pattern  scores 
X2  -  sum -of  square  of  pattern  score  .. 
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